AAN+: Generalized Average Attention Network for Accelerating Neural Transformer

Transformer benefits from the high parallelization of attention networks in fast training, but it still suffers slow decoding partially due to linear dependency O(m) decoder self-attention on previous target words at inference. In this paper, we propose a generalized average network (AAN+) aiming speeding up by reducing O(1). We find that learned weights follow some patterns which can be approximated via dynamic structure. Based insight, develop AAN+, extending our previously proposed (Zhang et al., 2018a, AAN) support more general position- and content-based patterns. AAN+ only requires maintain small constant number hidden states during decoding, ensuring its O(1) dependency. apply as drop-in replacement selfattention conduct experiments machine translation (with diverse language pairs), table-to-text generation document summarization. With masking tricks programming, enables decode sentences around 20% faster without largely compromising training speed performance. Our results further reveal importance localness (neighboring words) capability modeling long-range